Maintenance windows for ECDN

• This release will contain updates to all three ECDN components. To use these new capabilities, existing ECDN server instances will need to be upgraded.
• You need to take three actions:
  • Ensure you have setup an recurring maintenance window in the ECDN portal. We will upgrade your ECDN servers at the specified time.
  • Setup an SSH-key in the ECDN portal. - We will sunset remote password based logins into ECDN servers on Thu May 2, 2019. Beyond this time you will only be able to login to the ECDN server from the hypervisor console. For any kind of remote access, you will need to setup an SSH-key in the ECDN portal.
  • You need to identify at least one member in your team to play the role of af Super Admin and alert IBM support team of this choice, so that they can assign to this person the Super Admin role. Read more about user role.
• The changes in this release are:
• Portal:
  • Added: User roles: Super Admin, Admin and Reader. All existing ECDN portal logins will be given Admin role provileges, which is the same as what you are able to do today. A Reader role will have read-only privileges within ECDN portal.
  • Added: User accounts: page that shows you a list of all logins created for your ECDN account. Access to this page is restricted to only those with Super Admin role.
  • Added: As a Super Admin you can add new ECDN portal login account for a new team member, or suspend access for someone who may have left your team.
  • Added: As a Super Admin you can view a list of recent logins for a given user with their timestamps and IP addresses.
  • Added: SSH-key based remote logins. All remote access to ECDN server instances will require you to first register your SSH-key in the ECDN portal. Password based remote logins will be restricted to only hypervisor console based logins after the initial transition time.
  • Added: On Server list page a new batch action to allow you to enable or disable remote access on one or more ECDN server instances in a single operation.
  • Added: On Server list page a new batch action to allow you to copy the names or IP addresses of the ECDN servers.
• Virtual Appliance
  • Added: Support for SSH-key based remote logins.
  • Added: This version will include all known operating system security patches upto the release date.
• Backend services
  • Added: APIs for the new functionality described above.

• During this maintenance window, playback will not be affected. Access to portal will not be affected.

• This is the third and final phase of the maintenance window release rollout.
• Portal:
  • Changed: A new version number scheme for ECDN servers. Major. Minor. Sub-minor (Date). The date reflects the most recent date up to which all known operating system security patches that have been installed.
  • Changed: After login, the first page displayed will be the servers list page. Previously, the locations list page was displayed. This will allow you to right away see the status of all the ECDN servers.
  • Changed: In the top menu bar, first item will now be Servers (now a dropdown widget). Locations has been moved to the right.
  • Added: New "Software updates" menu item under "Servers" dropdown in the top menu bar. This page will allow you to set a recurring maintenance window for automated server upgrades. You can also see a list of all the servers that are not running the latest version along with a status indicator of whether they meet the required connectivity requirements.
  • Added: On servers details page, new status indicators to reflect the health and connectivity of the ECDN servers. The additions are: Remote assistance connections and NTP server connections. Existing status indicators have been enhanced to show the results of new self-tests for ensuring this server: has a valid SSL certificate, has a stable connection to upstream content server (midgress or parent server), and a stable connection to the Software update sites.
  • Fixed: For locations, setting a parent node or specifying an upstream midgress regions has been merged into a single setting. For the "Upstream source" value, you can now pick a midgress region or a parent ECDN location.
  • Fixed: Reports request page does not correctly account for DST changes.
  • Fixed: Changing midgress regions in the portal did not trigger server configuration updates.

• During this maintenance window, playback will NOT be affected. There may be momentary outages while accessing the portal.

• This is the second phase of the automated maintenance window release rollout. In this phase, like before, we will ask you one more time for a maintenance window to upgrade all the existing ECDN server instances to the latest version. After this server upgrade, all subsequent upgrades can be done as per an automated schedule.
• Virtual appliance:
  • Added: Support for running upgrade scripts as per a specified schedule.
  • Added: New self-test scripts added to check test NTP server connectivity and clock drifts. The results will be displayed in the portal as health status indicators.
  • Added: New self-test scripts added to check connectivity to all end points required to install software upgrades. The results will be displayed in the portal as health status indicators.
  • Fixed: Remote assistance VPN connectivity is checked to have successfully established a connection before returning a success result. This fix will make the status display in the portal more reliable.
  • Fixed: Cache file system out-of-space issue.

• During maintenance window you specified, while your servers are being upgraded, all playback will be affected. Access to the portal will not be affected. You will see gaps in the performance metrics for the duration of this upgrade window.

• To simplify the process around installation of security patches and ECDN server version upgrades, you will now be able to define a recurring maintenance window in the ECDN portal. At the designated time, automated scripts will be run to install the updates. This rollout will be done in three phases.
• Phase 1: Backend upgrade - Today.
• Phase 2: Update all customer ECDN server instances - January 10 to January 18. See below for details.
• Phase 3: Portal upgrade - January 22. See below for details.
• Backend services:
  • Added: Support for automated ECDN server upgrades.
  • Fixed: Changing midgress regions in the portal did not trigger server configuration updates.

• During this maintenance window, playback will be affected for all customers for up to 20 minutes. Access to the Portal will not be affected.

• We will replace the SSL certificates used to secure network to the ECDN Management portal and ECDN backend services end points.
• For more information, see this link.

• During this maintenance window, playback will NOT be affected. There may be brief up to 20 minutes outage while accessing the Management Portal or the ECDN servers backend services. As a result, you may see gaps in the performance metrics.

• Manual control of what ECDN servers players can connect to. This is an optional feature. By default, the video player will connect to the "closest" ECDN server, which is determined dynamically. With this feature, for each location, you will have the option to specify a whitelist or blacklist of local IP subnets. When enabled, the ECDN servers will only serve video content to players in those local subnets. You will manually have to update local subnet list in the ECDN portal, as subnets are added or removed in your local network. For more details see this support article.
• A ECDN server upgrade is required to version: 2.1 .20180806123507 to use this capability.
• Other changes:
  • [portal] only displays a subset of long External IP lists on Edit Location page instead of the full list. A Show more link is added to be able to perceive the entire list.

• During this maintenance window, there will be no outage in the Management Portal or ECDN servers. The backend will have a brief downtime. There may be gaps in the performance metrics for up to 15 minutes. Playback will NOT be affected.

• Portal:
  • New "Reports" page allows you to submit a request for retrieving usage metrics. On submission, you will get a request ID. Your request will be processed asynchronously and when ready, will be emailed to the address specified.
• Backend:
  • Metrics: Now you can request usage data to be aggregated by the hour. Previously, the API calls only returned data aggregated by the minute. Minute aggregation is only available and applicable for report duration of less than 24 hours. For report duration exceeding 24 hours, the data can only be aggregated by hour.
  • Changed: Modified the email template used for usage reports results. Make it explicit that users should NOT reply to the email address, but instead open a support ticket if they have additional questions.

• During this maintenance window, there will be no outage in the ECDN servers. The Management Portal and backend will have a brief downtime (< 1 minute). There may be gaps in the performance metrics for up to 15 minutes. Playback will NOT be affected.

• In this maintenance window we will be migrating the ECDN infrastructure from legacy data center to the IBM data center.
• During this time, there will be:
  • No outage in the delivery of video. All existing ECDN servers will continue to deliver content.
  • For up to 1 hour, you will not be able to access the ECDN Management Portal.
  • The ECDN Backend services will not be able to record usage metrics, and there will be gaps in collected usage metrics.
  • You will need to update the firewall settings to also allow network connectivity to the new IP address ranges 169.61.20.80/29 and 169.60.100.248/29, as documented in this support article. You will have until Jun 15th to make this firewall change.

• This maintenance window is required to make key infrastructure changes that support the ECDN backend services. No action is needed on customers side.
• During this maintenance window, there WILL be outage of up to 30 minutes, in the Management Portal and ECDN backend. Playback will be affected too. There will be gaps in the performance metrics.

• Virtual appliance:
  • A new version of the virtual appliance will be released to make ECDN servers compliant with GDPR regulations. We strongly recommend that all new installations of ECDN servers use this version going forward.
  • All existing ECDN server instances will need to be upgraded to bring them into compliance. All upgrades MUST be completed by May 15, 2018 in order to be compliant.
  • No actions are needed from customers side. IBM ECDN team will do these upgrades. IBM Video Streaming support team will work with each customer to identify a mutually convenient 3 hours maintenance window to do these ECDN server upgrades. During the upgrade, there will be downtime. Each server is expected to be down for less than 5 minutes. All ECDN servers belonging to one customer will be upgraded within a single maintenance window.
• Other changes:
  • [backend] Fixed. Now location configuration changes are first saved in the ECDN database, before updating the other backend SDCDN service. There is no longer any timing related ambiguity in the net outcome. As a result, after making updates to the external IP address settings for a location in the ECDN Management portal, users now no longer have to disable/enable each server in that location for the changes to go into effect.

• During this maintenance window, there will be no outage in the Management Portal or ECDN servers. The backend will have a brief downtime. There may be gaps in the performance metrics for up to 15 minutes. Playback will NOT be affected.

• [backend] Added new internal metrics collection API that writes the received data to a new data store that has a 2 year retention policy.
• [backend] Added a customer facing API that allows users to retrieve ECDN usage metrics for any duration in the past two years. In a subsequent release, an UI front end to submit and manage these requests will be provided.
• Existing ECDN server instances:
  • A configuration change is required to enable the server instance to upload usage metrics to the new API backend that can retain the data for 2 years.
  • ECDN operations team will trigger this update on all existing servers. You do NOT have to initiate any actions. There will be an outage of < 30 seconds on each ECDN server.
• Portal:
  • Fixed: Servers details page - Parent location & upstream servers section. Now the display of the value does NOT have a distracting flicker on page load. Default text is suppressed. Only when the asynchronous API returns, the value is updated.

• During the maintenance Management Portal will experience brief outages for < 5 minutes. There may be gaps in the performance metrics for up to 15 minutes. Playback will NOT be affected.

• Portal:
  • Servers page now additionally displays total number of viewers and number of viewers buffering for each server.
  • Servers page now allows users to filter list of servers by buffering ratio.
  • Support & Downloads page now enables users to copy version number of a virtual appliance by clicking the displayed value in the list.
  • Servers page, and Support & Downloads page now show a new notification alert when servers need to be upgraded. This notification contains a link to a new page where servers requiring an upgrade are listed.
  • Support & Downloads page - updated the content and layout with links to new support articles.
  • Location details page now refreshes the list of servers and their status every 5 seconds.
  • Servers page and Location details page now truncates longer server names. Full name is displayed as tooltip on hover.
  • Midgress region was not properly saved after editing a location in certain edge cases.
• [backend] New API to return the list of servers that are not running the latest version and need to be upgraded.

• During the maintenance Management Portal will experience brief outages for < 5 minutes. There may be gaps in the performance metrics for up to 15 minutes. Playback will NOT be affected.

This is a hotfix release containing fixes to address defects.

• Fixed: Locations details page does not display second or more external IP addresses. All external addresses were seen only in the Locations list page.
• Fixed: Edit locations page only saves the first external IP address. Second and more addresses are ignored.
• Fixed: VOD playback was negatively affected on those ECDN servers which had VOD cache persistence to disk enabled, and had the cache disk full.
• Fixed: In certain edge conditions users experienced errors while generating the virtual CD configuration file when editing a server definition in the Management Portal.

• During the maintenance Management Portal will experience brief outages for < 5 minutes. There may be gaps in the performance metrics for up to 15 minutes. Playback will NOT be affected.

• Persist recorded videos contents in ECDN server cache to disk. Previously all cache content was in memory, and was lost after a server reboot. Existing ECDN server instances will need to be upgraded to enable them to use this new feature. Upgrade will require the ECDN servers to be rebooted. ECDN operations team will do this upgrade and reboot. This is an optional feature. Even after existing ECDN servers are upgraded with this new release, you will need to make additional changes manually on the host hypervisors to enable cache persistence to disk. You don't have to immediately do the manual steps. You may delay them as per your convenience and when you are ready to use this capability.
  • All contents in the cache disk will be encrypted .
  • Only recorded video playbacks are cached to disk on first use. Live broadcast stream contents are cached only in memory.
  • Minimum recommended size of the cache disk is 4GB.
  • Management Portal displays on the server configuration page new controls to enable or disable the use of cache disk. When disabled, recorded videos content will also be cached in the memory.
  • Management Portal displays on the server details page new graphs showing the usage metrics of the cache disk.
• Hardened the ECDN server and fixed issues identified during the vulnerability scans.

• During the maintenance Management Portal will not have any downtime. There may be gaps in the performance metrics for up to 15 minutes.

• We have now released our own content distribution solution that is dedicated for content delivery to the ECDN server instances. It includes set of “midgress” servers in select IBM Video Streaming data centers around the world. ECDN server instances will now connect to the midgress server in their region to get the video content. This eliminates the need for enterprises to open firewall rules to allow connectivity to 3rd party CDN providers.

  To see the benefits of these midgress servers, we will need to update the configuration settings on all existing ECDN server instances. In this update, the upstream server will be changed from the 3rd party CDN provider to the closest IBM midgress server.

  Once you give us an update maintenance window, we will trigger a batch update on all your server instances. It will take about 30 seconds to refresh the configuration for each instance.

• ECDN Management portal will not have any downtime.
• There may be gaps in the performance metrics for up to 15 minutes.

• ECDN Management portal will not have any downtime.
• There may be gaps in the performance metrics for up to 15 minutes.

• Streaming of live or VOD content will NOT be affected.
• ECDN Management portal will not be accessible for 30 minutes.
• There may be gaps in the performance metrics for up to 30 minutes.