CrafterCMS

Integrate CrafterCMS with Jenkins to Automate DevOps: Code Forward, Content Back Process

Great DevOps helps us build better software products faster. One of the key elements of DevOps is automation within the development process across lower development environments.  Jenkins, Bamboo, Travis and many other platforms like them are used by DevOps teams to help automate the process of Continous Integration and Continous Delivery (CI/CD). To a large degree, real support for CI/CD and agile development is something that is woefully missing in Content Management, a core component of digital experience platforms.  The toolset and the architecture of today’s content management platforms make it difficult, sometimes near impossible to support let alone automate. How easy is it for your team to take content from the Production CMS and update content in lower environments like Dev, QA?  How easy is it to roll out new features?  Often times this process is not only laborious for the DevOps team, it also halts the content authoring process. CrafterCMS’s, an open source CMS Git-based repository and Java/Spring and Groovy-backed stack are a game changer in this regard.  Further, Crafter integrates directly into to your CI/CD process.  In this article, using Jenkins as our example we’ll demonstrate how you can connect your production CMS to your developer workflow to facilitate Code Forward, Content Back(TM) workflow. We call this seamless collaboration between devops and authors DevContentOps®.

You can read more about this process in detail here:

Although our example is based on Jenkins, the scripts and flow used in this blog are applicable to nearly any automation

Understanding Where Code Forward, Content Back Automation Fits

The fact is that every step of your DevOps process is open to automation.  In this section, we’ll cover the common points of integration, specifically:

  • The point in the process where you want to move content from your production CMS “back” to lower environments to support development and testing.
  • The point where you want to promote code “forward” from the development process to the production CMS so it can be published.

Both of these points in the process are illustrated and addressed in the diagram below by the double-headed arrow labeled Code Forward, Content back.   With respect to the CMS, Code Forward, Content Back is the most important aspect of the DevOps automation.

CrafterCMS’s Git-based repository is the foundation of the automation.  Our automation running in Jenkins is going to leverage API’s within the authoring environment (Crafter Studio) to sync code and content with the development process.  APIs will also be used to publish code synced to authoring to the Production delivery infrastructure.

Implementing Code Forward, Content Back Sync

Now that we’ve established what integration we’re addressing here is, let’s focus on configuring it.  Take a lookat the diagram below, this elaborates the previous diagram showing how the sync occurs.

Note that both the Production authoring and the Development “environment” has a repository.  In authoring, this is a local Git repository.  In development, this is most often a centrally hosted Git repository that supports workflow and review (like Bitbucket, Gitlab, Github, and others.)   You can think of the repository under authoring as the Content Repository and the repository supporting developers as the Code Repository.  These names (Content Repository, Code Repository) are simply labels to help describe their purpose and assist us in addressing them in the context of this article.

To facilitate this flow, the Content Repository under authoring/Crafter Studio declares the Code Repository as a remote. The primary way of “syncing” work between Git repositories is through pull and push operations.  Before you can push your work to a remote, you must first pull merge the updates (if any) from the remote.  Once done, you can push your changes to the remote.

Automating the Pull / Push of Code and Content

To help automate the process described above Crafter Studio, the authoring and repository component of CrafterCMS supports a set of APIs.  You can find a full listing of Crafter Studio APIs for Crafter 3.0 here: http://docs.craftercms.org/en/3.0/developers/projects/studio/index.html

These APIs are easily invoked by a script.   You can use the following example script in your own implementation:

codeforward-contentback-sync.sh

#!/usr/bin/env bash
studioUsername=$1
studioPassword=$2
studioserver=$3
project=$4
remote=$5
branch=$6

echo "Authenticating with authoring"
rm session.txt
curl -d '{ "username":"'$studioUsername'", "password":"'$studioPassword'" }' --cookie-jar session.txt --cookie "XSRF-TOKEN=A_VALUE" --header "X-XSRF-TOKEN:A_VALUE" --header "Content-Type: application/json"  -X POST $studioserver/studio/api/1/services/api/1/security/login.json

echo "Pull from remote (get code waiting to come to sandbox)"
curl -d '{ "site_id" :"'$project'", "remote_name":"'$remote'", "remote_branch":"'$branch'" }' --cookie session.txt --cookie "XSRF-TOKEN=A_VALUE"  --header "Content-Type: application/json" --header "X-XSRF-TOKEN:A_VALUE" -X POST $studioserver/studio/api/1/services/api/1/repo/pull-from-remote.json

echo "Push to remote (send content waiting to go to development)"
curl -d '{ "site_id" :"'$project'", "remote_name":"'$remote'", "remote_branch":"'$branch'" }' --cookie session.txt --cookie "XSRF-TOKEN=A_VALUE"  --header "Content-Type: application/json" --header "X-XSRF-TOKEN:A_VALUE" -X POST $studioserver/studio/api/1/services/api/1/repo/push-to-remote.json

Use of the script:

codeforward-contentback-sync.sh [USERNAME] [PASSWORD] [AUTHOR_SERVER_AND_PORT]  [SITE_ID] [REMOTE_NAME] [BRANCH_NAME]

USER_NAME is the Studio user (application account)
PASSWORD is the Studio user password (application account)
AUTHOR_SERVER_AND_PORT the protocol server name and port of Studio
SITE_ID the ID of the site
REMOTE_NAME the name of the upstream (typically origin)
BRANCH_NAME the name of the branch (typically master)

Example:
codeforward-contentback-sync.sh devops mydevopspw http://localhost myprojectID origin master

The script is quite simple.  It authenticates to Crafter Studio, performs a pull from the Remote Code Repository and then if there are no conflicts, performs a push.  These two operations move code updates forward to the production Sandbox (not yet live) and content back to the development process.  Only approved code that’s been moved to the “master” branch with the intention to release is moved forward.

Calling the Script in Jenkins

The first step is to create a project.  Give the project a clear name and select the Freestyle project then click OK to continue.

There is no Source Code Management (SCM) aspect of the project.  The most typical use case for Content back workflow is a scheduled event: Every hour, day, week etc.

The next step is to define build triggers.  Since you are calling APIs here and content back is most likely based on some schedule you define you want to indicate that there is no Source Code Management (SCM) aspect of the project.

Select “Build Periodically” and define your schedule.  Schedule definitions user standard Cron/Quartz configuration.

Finally, you must define that you want Jenkins to call your script:

Once you have done these steps you are ready to go.  Manually invoke this build any time you want directly through the Jenkins console.  I recommend testing it to make sure your parameters and schedule are correct.

Publishing Code That’s Been Sync’d to Sandbox

When you run the code forward, content back process code in the remote code repository is moved to the production authoring sandbox (content repository.)  This code is now staged for publishing.  It is not yet live.  Crafter Studio must publish the code, making it available to your delivery servers.  This in-and-of-itself is awesome: global, elastic deployment at the touch of a button.

So how is it done?  Crafter Studio provides an API that allows you to publish commit IDs.  You can provide a single commit ID or you can provide a list.  It’s typical as part of your release process to “Squash” all of the commits in a given release into a single commit ID.  This allows you to address all of the work as a single ID/moniker which makes it very easy to move, publish and roll back without missing anything.

These APIs are easily invoked by a script.   You can use the following example script in your own implementation:

publish-code.sh

#!/usr/bin/env bash
studioUsername=$1
studioPassword=$2
xsrf=AUTOMATED
studioserver=$3
project=$4
env="Live"
commit=$5

echo "Authenticating with authoring"
rm session.txt
curl -d '{ "username":"'$studioUsername'", "password":"'$studioPassword'" }' --cookie-jar session.txt --cookie "XSRF-TOKEN=A_VALUE" --header "X-XSRF-TOKEN:A_VALUE" --header "Content-Type: application/json"  -X POST $studioserver/studio/api/1/services/api/1/security/login.json

echo "Publishing Commit $commit"
curl -d '{ "site_id" :"'$project'", "environment":"'$env'", "commit_ids": ["'$commit'"] }' --cookie session.txt --cookie "XSRF-TOKEN=A_VALUE"  --header "Content-Type: application/json" --header "X-XSRF-TOKEN:A_VALUE" -X POST $studioserver/studio/api/1/services/api/1/publish/commits.json

Use of the script:

publish-code.sh [USERNAME] [PASSWORD] [AUTHOR_SERVER_AND_PORT]  [SITE_ID] [COMMIT_ID] 

USER_NAME is the Studio user (application account)
PASSWORD is the Studio user password (application account)
AUTHOR_SERVER_AND_PORT the protocol server name and port of Studio
SITE_ID the ID of the site
COMMIT_ID the squashed commit ID of the items coming from the release branch

Example:
publish-code.sh devops mydevopspw http://localhost myprojectID 378d0fc4c495b66de9820bd9af6387a1dcf636b8

The script is quite simple.  It authenticates to Crafter Studio and invokes a publish for the provided commit.  This op

Calling the Script in Jenkins

See configuration of sync script above.  The steps are exactly the same with the following differences:

  1. You will call the publish-code script instead of the codeforward-contentback script.
  2. You will ask the user for a parameter  value COMMIT_ID via the UI on each invocation and pass that to the command line as the COMMIT_ID parameter value

 

That’s it!  You can now publish your code releases via commits to your entire delivery infrastructure regardless of its size or distribution.

Conclusion

CMS platforms are notorious for refusing to play nice with CI/CD and agile development practices and process, automation and tools like Jenkins, Bamboo, Travis and others.  Databases and JCR repositories are one component of several fundamental, architectural limitations that make supporting CI/CD difficult for CMS platforms. Crafter is an open source, dynamic CMS with a unique Git based repository specifically designed to fit neatly in to your development practices and bring your authoring and development teams together in a way never before possible to improve and increase the rate and volume of innovation!