See Kaltura Install Packaging Project for the parent project.

The Continuous Integration system will be responsible for the automation of the following tasks:

• Overall build of the Kaltura platform packages against the master branch, release branches and approved pull-requests (for both nightly and stable releases).
• Pushing the packages to the install repositories.
• Perform a full Kaltura deployment on a test cluster.
• Perform automated testing of the installed server features via API calls and command line scripts, determining overall build stability for both clean install and version to version upgrades.
• Generate web-page build reports and email in case of fails.
• Distribute packaged/compiled API client libraries to respective language repositories.

An automatic approach to the build, test and release process has many advantages, most prominently, it significantly reduces the time to release new packages and verifies that packages were fully tested before being used in production. We’ve also listed some key advantages in our specific project -

• Release more often, faster, and provide nightly builds for advanced platform testers.
• Ensure new commits do not break existing functionality on the master branch.
• Allow contributors to make changes and additions with a higher level of security, knowing pull-requests are tested as part of the whole system in production mode, before being merged.
• Provide elaborate platform test reports before entering official manual QA phase.

This will be the general flow of each CI run.

1. Versions of packages included in the new release will be updated in spec files.
2. Additional required steps for the upgrade [SQL alter scripts, additional token replacements, etc] will be added to the postinst according to need
3. The packages will be built
4. Packages are uploaded to the repository using an SSH key
5. Packages are distributed to Akamai

Following successful build, AMI instances are launched via EC2 CLI API, and Kaltura will be deployed on them.
All deployments will be done using self-signed SSL certificates.

• 2 fronts behind an HTTPs LB (a Linux machine running Apache configured to use the ProxyPass module)
• 1 Admin Console front
• 2 batch servers
• 2 sphinx servers
• 1 MySQL DB server (shall we also test replication?)

• 2 fronts behind an HTTPs LB (a Linux machine running Apache configured to use the ProxyPass module)
• 1 Admin Console front
• 2 batch servers
• 2 sphinx servers
• 1 MySQL DB server (shall we also test replication?)

Following successful deployment and upgrade of the Kaltura clusters, the test suites will be ran, and build & test reports will be generated.

All API calls and apps will be loaded over SSL.

The following test cases will be run in the following order, on each cluster deployment (both clean and upgrade). Nightly testing should run a complete regression coverage via API client libs, verifying the stability of the latest MASTER branch.

1. Check space on / partition
2. Check space on web partition
3. Check KDP3 version is correct by comparing KMC's config.ini with the actual last created KDP3 dir on disk.
4. Check KMC version is correct by comparing KMC's config.ini with the actual last created KDP3 dir on disk.
5. Check KDP3 version is correct by comparing KMC's config.ini with the actual last created HTML5 dir on disk.
6. Verify that all relevant processes (Apache, MySQL, Sphinx, batch, memcache, monit) are up and running on all machines in the cluster
7. Verify that all processes and crons are properly configured to run after system restart
8. Verify HTTPs call redirects for start page, KMC, Admin Console and testme. Perform curl request (with redirect follow) to each of the URLs, and test the response returned as expected:
   1. https://[DOMAIN]/ --- Verify Start Page
   2. https://[DOMAIN]/api_v3/testme/ --- Verify TestMe Console URL
   3. https://[DOMAIN]/index.php/kmc --- Verify KMC URL
   4. https://[DOMAIN]/admin_console/ --- Verify Admin Console URL
   5. https://[DOMAIN]/apps/studio --- Verify Universal Studio URL
9. Verify system restart behaviour (run 1 through 3 post restart)
10. Verify that processes (Apache, MySQL, Sphinx, batch, memcache) are being relaunched by monit after MANUAL kill (testing crash resurrection).
11. Verify new publisher account creation. Continue all following tests on this new partner account.
12. Verify profile ID creation.
13. Verify profile ID delete.
14. Test email logs for sent new publisher account activation email.
15. uiConf and file verifications -
    1. Run through all the uiConfs in the database.
    2. For each uiConf, run through the uiConf object URLs AND inside the uiConf XML for all referenced file paths (swf, js, image files, etc.) and verify the existence of these files on disk.
    3. Check the kmc.swf and login.swf requests return 200
16. Verify simple transcoding: Upload video, see complete transcoding flow finished successfully.
17. Verify reconversion: reconvert an existing asset, see complete transcoding flow finished successfully.
18. Verify fallback transcoding: Rename the ffmpeg symlink. Upload video, see complete transcoding flow finished successfully. Rename the ffmpeg symlink back.
19. Verify clipping and trimming API
20. Verify clipapp index page loads
21. Run all client libraries and their respective unit-tests
22. Enable dropfolder plugin
23. Create a Local DropFolder
24. Test Bulk Upload XML that includes custom metadata fields and thumbnails.
25. Setup email notification for new entry event
26. Setup HTTP notification for entry change event
27. Create a new Entry
28. Check mail logs to see if email was sent.
29. Analytics verification
30. Upload captions
31. Search string in captions and entry metadata
32. Check the report API to see the play count
33. Curl playManifest to that Entry WITH a valid sview KS, see that the video returns
34. Thumbnail API verification:
    1. This test will have a stored prepared image to compare against
    2. Call the thmbnail API for second 2 of the new entry
    3. Use ImageMagick to compare the returned image against the stored test imaged
35. Verify Red5 is working - http://exchange.nagios.org/directory/Plugins/Software/FMS-monitor/details
36. Verify YouTube distribution (create profile, distribute entry, query for success)
37. Check the bandwidth report API to see bandwidth and storage counts
38. Verify KS Access Control:
39. Create an AC Profile with KS protection
40. Assign it to a Video Entry
41. Curl playManifest to that Entry without a KS, see that the video fails to return
42. Create a Local XML DropFolder, copy a file to the folder, test the file was successfully pulled in to Kaltura, transcoded and that the XML metadata exists.
43. Create a Remote Storage profile against an S3 Bucket, verify that content uploaded gets pushed to the S3 bucket.
44. Verify Player - Use http://phantomjs.org/ to run base tests against player embed, playlist embed, thumbnail embed, and common player scenarios (play, pause, seek)

For each step in the CI cycle, execution time measurements will be performed and saved in order to analyze platform trends over time. The following CI steps will measured:

1. Time it took to pull the code from git repositories.
2. Time it took to build packages.
3. Time it took to push packages to install repositories.
4. Time it took to install each package on the test clusters (clean and upgrade).
5. Time it took to run post-inst scripts per package.
6. Time to run each unit-test.
7. Aggregate time from pulling code till finish tests (complete cycle).

• Full test report available on a URL with the version-date combo.
• Header should show overall health status of the build - Percentage of fail/pass
• Per test, status (FAILED or PASSED), and if failed, show unit test error output. According to the following table:

Setup a mailing list for people to subscribe for reports via email. 3 types of emails:

1. If RPM layer failed packaging - email subject: [PACKAGING_FAILED] Kaltura Release - {CORE.VERSION} - {BUILD.DATE}
2. If any of the core unit tests failed - email subject: [CORE_FAILED] Kaltura Release - {HEALTH.PERCENT} - {CORE.VERSION} - {BUILD.DATE}
3. If all tests passed successfully - email subject: [BUILD_READY] Kaltura Release - {CORE.VERSION} - {BUILD.DATE}

The body of the email will be a full table report of the test suite (as defined above in web reports).

• Nightly packages are built and saved into the /nightly/ folder.
• Pre-QA-Release packages are built and saved into the /release/ folder with respective release version and date.

• If Pre-QA-Release passes 100%, also distribute packaged/compiled client libraries to respective repositories: http://rubygems.org/ , https://www.npmjs.org/ , https://getcomposer.org/ https://pypi.python.org/

All code in this project is released under the AGPLv3 license unless a different license for a particular library is specified in the applicable library path.

Copyright © Kaltura Inc. All rights reserved.

Authors @jessp01, @zoharbabin and many others.