Credits : https://sunitc.dev/2020/01/21/spring-boot-how-to-add-jpa-hibernate-envers-auditing/

1.0 Introduction

In our previous blogs, we demonstrated how to create Spring Boot application and add Flyway migration, and implement Authentication/Authorization.

In this blog, we will look at how to add Auditing feature to the application.

So the question is “What is Auditing and why is it important?”

In any application, auditing means that we track and log every change in all business objects, i.e, track each insert, update and delete operations.
Basically it involves tracking three things

• What operation was performed?
• Who did it?
• When was it done?

Auditing helps us in maintaining history records, which can later help us in tracking user activities.

You can follow the steps in this blog, using any Spring Boot application. However for this blog, we will start with the base application that we build in our previous blog, which has flyway integration and authentication build into it.
You can find the starting base code for this blog in – https://github.com/chatterjeesunit/spring-boot-app/tree/v3.0

We already have a Customer Entity in our application, and we will add auditing to this entity.

Top ∆

2.0 JPA Auditing

This is the simplest form of auditing.
Here we do not track what are the changes done, but only keep track of who created or modified a business entity and when it was done.

To be specific, we will keep track of following additional fields for each business object

• Created On
• Created By
• Modified On
• Modified By

2.1 Adding new audit columns to database

Let’s start by adding the Created/Modified columns into the database for Customer table.

ALTER TABLE `customer`
ADD COLUMN `created_by` VARCHAR(50) NOT NULL,
ADD COLUMN `updated_by` VARCHAR(50) NULL,
ADD COLUMN `created_on` DATETIME NOT NULL,
ADD COLUMN `updated_on` DATETIME NULL;

You can either run the below script directly on your database, or you can add it to your Flyway/Liquibase scripts (in case you are using them or any other such database versioning tools).

2.2 Adding audit fields to entity class

Now that we have added the fields to the database, let’s add the fields to the our Customer Entity.
Add following code.

    @Column(name = "created_by")
    @CreatedBy
    private String createdBy;

    @Column(name = "updated_by")
    @LastModifiedBy
    private String updatedBy;

    @Column(name = "created_on")
    @CreatedDate
    private Date createdOn;

    @Column(name = "updated_on")
    @LastModifiedDate
    private Date updatedOn;

• Annotations : We have added annotations from org.springframework.data.annotation to all the audit fields
  • @CreatedBy /@LastModifiedBy : Declares a field as the one representing the principal that recently created / modified the entity containing the field
  • @CreatedDate / @LastModifiedDate : Declares a field as the one representing the date the entity containing the field was created / recently modified.
• Date Fields
  • In our example we have used java.util.Date
  • The field can be any of the following types – Joda-Time, DateTime, legacy Java Date and Calendar, JDK8 date and time types, and long or Long.

If you don’t want to add these fields and annotations on each of your entity class, your entity can also extend your entity from org.springframework.data.jpa.domain.AbstractAuditable

2.3 Populating the logged in user info

The created and modified dates will be automatically populated by JPA.

However we have to provide details on what to persist in the created_by and updated_by columns.

2.3.1 Implementing AuditAware<T>

The first thing that we have to do is implement AuditAware<T>, where T is the type of CreatedBy/ModifiedBy field (which for us is String)

• Create a new class AuditAwareImpl that implements AuditAware<String>
• Implement the method – public Optional getCurrentAuditor()
• Add logic to the method getCurrentAuditor(), to return the username of the current logged in user
  • We can fetch the current SecurityContext, fetch the current Principal from it, and then get user details from it.

public class AuditAwareImpl implements AuditorAware<String> {
    @Override
    public Optional<String> getCurrentAuditor() {
        return Optional.ofNullable(SecurityContextHolder.getContext())
                .map(SecurityContext::getAuthentication)
                .filter(Authentication::isAuthenticated)
                .map(Authentication::getPrincipal)
                .map(User.class::cast)
                .map(User::getUsername);
    }
}

2.3.2 Enable JPA Auditing

Next we will create a Configuration to enable JPA Auditing and also provide an implementation for the AuditAware<String> bean.

• Create a new Config class
• Add annotation @EnableJpaAuditing to this class
• Create a method to return Bean for AuditorAware<String> – and return a new instance of the AuditAwareImpl class that we had created in previous step.

@Configuration
@EnableJpaAuditing(auditorAwareRef = "customAuditProvider")
public class JpaConfig {

    @Bean
    public AuditorAware<String> customAuditProvider() {
        return new AuditAwareImpl();
    }
}

Last configuration is to associate an EntityListener class to your audited entity. Spring already provides one such listener for us – AuditingEntityListener

@Entity
@EntityListeners(AuditingEntityListener.class) // add this line
public class Customer {
  ....
  ....
}

2.4 Testing JPA auditing.

Now that we have added all the required implementation, its time to test it.
We will test using Rest client curl.

The examples used . or testing you have authentication/authorization already configured in the spring boot app, and it has two users in the database

– admin01@tw.com (password: admin01@123#)
– admin02@tw.com (password: admin02@123#)

However these users and not mandatory.
You can test with your own set of users, by changing the curl statement in below examples.

Login as and user (admin01@tw.com) and create a customer using below command

curl -i -u admin01@tw.com:admin01@123# -X POST -H "Content-Type: application/json" http://localhost:8080/customer/ -d '{"firstName":"John","lastName":"Doe","emailAddress":"johndoe@gmail.com","addresses":[{"streetAddress":"4487 Elsie Drive","city":"Onida","stateCode":"SD","zipCode":"57564","country":"USA"}]}'

Now login as another user (admin02@tw.com) update the same customer, and change last name . (Change the id at end of PUT url to match the id of the created customer)

curl -i -u admin02@tw.com:admin02@123# -X PUT -H "Content-Type: application/json" http://localhost:8080/customer/1 -d '{"firstName":"John","lastName":"Doe01","emailAddress":"johndoe@gmail.com","addresses":[{"streetAddress":"4487 Elsie Drive","city":"Onida","stateCode":"SD","zipCode":"57564","country":"USA"}]}'

Go to database and run sql – select * from customer where email_address = 'johndoe@gmail.com'

Top ∆

3.0 Hibernate Envers Auditing

Although JPA Auditing provided easy configuration for basic auditing, it only provided following information

• When was an entity created and who created it.
• When was the entity last modified and who modified it.

It does not provides you with details of all the changes/updates that were done for an entity. e.g a Customer entity could have been modified 5 times. With JPA auditing, there is no way i can find out what was changed in the entity in each of the 5 updates, and who did them

And so Hibernate Envers comes into the picture, and it provides complete Audit History for an entity.

3.1 How does Hibernate Envers stores audits in database

• Envers creates a REVINFO table
  • This table is common across all entities in the application.
  • It contains details of when each revision of entity was done and by whom
  • By default it contains two columns
    • REV – Primary Key Column – Revision ID/ Revision Number
    • REVTSTMP – Timestamp of when Revision was created.
    • Additionally we can also store user information also in this table in a custom column
• Basically, Envers creates an ‘AUDIT‘ table for each entity that needs to be audited.
  • e.g if we want to audit the changes to CUSTOMER table, Envers will create store the audit history in a CUSTOMER_AUD table. (Name of the table is suffixed with _AUD)
  • This *_AUD table has all the columns as the main entity table.
  • In addition it also has a two other columns
    • REV : Revision ID – Foriegn key Reference to the REVINFO.REV
    • REVTYPE : Has three possible numeric values to denote the type of operation on entity – 0 (Add), 1 (Update), and 2 (Delete)

Hibernate provides some customizations to the name of the Enver tables and columns. We will do following customizations in our application for better readability of table and column names

• The table Suffix _AUD will be changed to _AUDIT. e.g the audit table for CUSTOMER will be now CUSTOMER_AUDIT. Following columns of all the audit tables will also be renamed
  • REV column will be renamed to REVISION_ID
  • REVTYPE column will be rename to REVISION_TYPE
• REVINFO table will be renamed to REVISION_INFO. Additionally following columns will be renamed in the revision table
  • REV column will be renamed to REVISION_ID
  • REVTSTMP will be renamed to REV_TIMESTAMP

To Summarize these is how the ER Diagram of AUDIT tables will look like.

You might have noticed that there is NO Foriegn key reference from audit tables to main entity tables for ID column.

The reason for this is that in some scenarios we may want to keep Deleted records also in the Audit tables. In such scenarios the foriegn key reference won’t work as records in main entity table will have been deleted.

3.2 Add auditing tables into the database

We have already seen how the audit tables and revision info tables will look like. Lets create those tables in the database.

You can either run the below script directly on your database, or you can add it to your Flyway/Liquibase scripts (in case you are using them or any other such database versioning tools).

CREATE TABLE `revision_info` (
    `revision_id` INTEGER PRIMARY KEY AUTO_INCREMENT,
    `rev_timestamp` BIGINT(20) NOT NULL,
    `user` VARCHAR(50) NOT NULL
);

CREATE TABLE `customer_audit` (
  `revision_id` INTEGER NOT NULL,
  `id` BIGINT(20) NOT NULL,
  `revision_type` TINYINT NOT NULL,
  `email_address` VARCHAR(255),
  `first_name` VARCHAR(255),
  `last_name` VARCHAR(255),
  `created_by` VARCHAR(50),
  `updated_by` VARCHAR(50),
  `created_on` DATETIME,
  `updated_on` DATETIME,
  PRIMARY KEY (`revision_id`, `id`),
  CONSTRAINT `idfk_customer_revinfo_rev_id`
	FOREIGN KEY (`revision_id`) REFERENCES `revision_info` (`revision_id`)
);

CREATE TABLE `address_audit` (
  `revision_id` INTEGER NOT NULL,
  `id` BIGINT(20) NOT NULL,
  `revision_type` TINYINT NOT NULL,
  `city` VARCHAR(255),
  `country` VARCHAR(255),
  `customer_id` BIGINT(20),
  `state_code` VARCHAR(255),
  `street_address` VARCHAR(255),
  `zip_code` VARCHAR(255),
  PRIMARY KEY (`revision_id`, `id`),
  CONSTRAINT `idfk_address_revinfo_rev_id`
	FOREIGN KEY (`revision_id`) REFERENCES `revision_info` (`revision_id`)
);

Top ∆

3.3 Enable Auditing

In this section, we will add auditing by fail first method.
This means we will add configurations/code one step at a time, start the application after that and see if application starts or fails. If it fails, we will fix it and repeat the cycle again till we get a working application.

This will help us understand why some configurations/code is required and what errors may come if we miss them.

3.3.1 Add dependency to build.gradle

The first configuration to be done is to add the Hibernate Envers dependency in build.gradle

compile('org.hibernate:hibernate-envers:5.4.25.Final')

3.3.2 Add @Audited annotation

Now add the @Audited annotation to all the entities that you want to audit

@Audited
public class Customer {
  ....
}

@Audited
public class Address {
  ....
}

3.3.3 Add configurations to application.yaml

Let’s try to start the application.

You will get following error

org.hibernate.tool.schema.spi.SchemaManagementException: Schema-validation: missing table [address_aud]

OR

org.hibernate.tool.schema.spi.SchemaManagementException: Schema-validation: missing table [customer_aud]

This is because Hibernate Envers is expecting Audit tables as *_AUD table.
However as you have seen, we have customized our table suffix as _AUDIT table. This means Envers is search for customer_aud table, address_aud table, but instead we have customer_audit and address_audit table. Additonally we had also customized the columns – REV, REVTSMTP and REVTYPE.

The failures are because we need to tell Hibernate Envers about these customizations. We can do so by adding the following properties in application.yaml

spring:
  jpa:
    properties:
      org:
        hibernate:
          envers:
            audit_table_suffix: _AUDIT
            revision_field_name: REVISION_ID
            revision_type_field_name: REVISION_TYPE

Now if you again try starting the application, you will get the below error

Schema-validation: missing table [customer_address_audit]

To Fix this error, add a @NotAudited annotation to the field addresses in the Customer entity class

    @NotAudited
    @OneToMany(fetch = FetchType.EAGER, orphanRemoval = true, targetEntity = Address.class, cascade = CascadeType.ALL)
    @JoinColumn(name = "customer_id")
    private List<Address> addresses;

3.3.4 Configure a RevisionEntity

Let’s try to start the application again.
This time you will get following error.

org.hibernate.tool.schema.spi.SchemaManagementException: Schema-validation: missing table [revinfo]

We have created a table REVISION_INFO instead of REVINFO.

However we cannot change the name of table via configurations in application.yaml. To do this we need to create a RevisionEntity class.

Envers already provides a class – org.hibernate.envers.DefaultRevisionEntity, and we will extend this class.

@Entity
@Table(name = "revision_info")
@RevisionEntity
public class AuditRevisionEntity extends DefaultRevisionEntity {

    @Column(name = "user")
    private String user;
}

• We created a class AuditRevisionEntity which extends from DefaultRevisionEntity.
• Add @Table annotation and provide name of table – revision_info
• Add annotation @RevisionEntity, which signifies that this is the RevisionInfo Entity for Hibernate Envers.
• Add a field for user, which we had added in the revison_info table.

Let’s try to start the application again. We will get following error this time around.

org.hibernate.tool.schema.spi.SchemaManagementException: Schema-validation: missing table [hibernate_sequence]

• Hibernate requires this for ID columns which have following annotation – @GeneratedValue(strategy = GenerationType.AUTO)
• Easiest way to solve this ito change the Generation Strategy from GenerationType.AUTO to GenerationType.IDENTITY
• However, this error is because of ID field in org.hibernate.envers.DefaultRevisionEntity, which we extended.

Since we cannot change DefaultRevisionEntity class, we will add below configuration in application.yaml to fix this.

spring:
  jpa:
    hibernate:
      use-new-id-generator-mappings: false

Let’s try starting the application again.
This time you will get following error, because Envers expects id and timestamp columns in the revision_info table.

Schema-validation: missing column [id] in table [revision_info]

Since we have renamed the columns to revision_id and rev_timestamp, we need to add configurations in our RevisionEntity class.

Just add following annotations to the AuditRevisionEntity class.

@AttributeOverrides({
    @AttributeOverride(name = "timestamp", column = @Column(name = "rev_timestamp")),
    @AttributeOverride(name = "id", column = @Column(name = "revision_id"))
})
public class AuditRevisionEntity extends DefaultRevisionEntity {
  ...
  ...
}

Now you should be able to start your application successfully.

Now you know what all different errors that came possibly come when you configure Envers auditing, and how to fix them.

Note: There is no need to add spring-data-envers library into your project. Hibernate Envers library is more than enough.

Top ∆

3.4 Populating Logged In User Info In Revision table

Now we have everything setup, and our application is also running.
Lets try to create a new customer.

curl -i -u admin01@tw.com:admin01@123# -X POST -H "Content-Type: application/json" http://localhost:8080/customer/ -d '{"firstName":"John","lastName":"Doe","emailAddress":"johndoe@gmail.com","addresses":[{"streetAddress":"4487 Elsie Drive","city":"Onida","stateCode":"SD","zipCode":"57564","country":"USA"}]}'

You will get following error

Caused by: java.sql.SQLException: Field 'user' doesn't have a default value

This happens, because Envers is trying to audit the data into revision_info table, and puts null into the user column.
This is because Envers does not know how to get the logged in user information and populate in this user column.

Let’s add code to add logged in user information.

• Create a class that implements org.hibernate.envers.RevisionListener
• In the new class implement the method – public void newRevision(Object revisionEntity)
  • In this method get the current user from Spring’s SecurityContext (similar to what we did in AuditorAwareImpl)
  • And set this user in AuditRevisionEntity

public class AuditRevisionListener implements RevisionListener {

    @Override
    public void newRevision(Object revisionEntity) {

        String currentUser = Optional.ofNullable(SecurityContextHolder.getContext())
                .map(SecurityContext::getAuthentication)
                .filter(Authentication::isAuthenticated)
                .map(Authentication::getPrincipal)
                .map(User.class::cast)
                .map(User::getUsername)
                .orElse("Unknown-User");

        AuditRevisionEntity audit = (AuditRevisionEntity) revisionEntity;
        audit.setUser(currentUser);

    }
}

Next add the following annotation to the RevisionEntity class – AuditRevisionEntity

@RevisionEntity(AuditRevisionListener.class)

3.5 Testing Hibernate Envers Auditing

Now let’s test the application and see if auditing is working correctly.

Login as and user (admin01@tw.com) and create a customer using below command

curl -i -u admin01@tw.com:admin01@123# -X POST -H "Content-Type: application/json" http://localhost:8080/customer/ -d '{"firstName":"John","lastName":"Doe","emailAddress":"johndoe@gmail.com","addresses":[{"streetAddress":"4487 Elsie Drive","city":"Onida","stateCode":"SD","zipCode":"57564","country":"USA"}, {"streetAddress":"2240 Bridge Parkway","city":"San Francisco","stateCode":"CA","zipCode":"94065","country":"USA"}]}'

Now login as another user (admin02@tw.com) update the same customer, and change last name and remove one of the address . (Change the id at end of PUT url to match the id of the created customer)

curl -i -u admin02@tw.com:admin02@123# -X PUT -H "Content-Type: application/json" http://localhost:8080/customer/1 -d '{"firstName":"John","lastName":"Doe01","emailAddress":"johndoe@gmail.com","addresses":[{"id":1, "streetAddress":"4487 Elsie Drive","city":"Onida","stateCode":"SD","zipCode":"57564","country":"USA"}]}'

Now let’s go to database and verify the audit history got populated correctly.

Run sql – select * from revision_info;
This will give you information on how many revisions were created and by whom.

Next look at the customer audit table – select * from customer_audit

You can see above that in Revision 1, the customer John Doe was created, and Revision 2, it’s last name was updated.

Finally let’s look at Address Audit table – select * from address_audit;

We can see from above that in Revision 1, two addresses were added. And in Revision 2, one of the address got deleted.

Deleted entries are stored in audit table as NULL values.
If you want to keep the old values instead of NULL, then set following property to true in application.yaml

org.hibernate.envers.store_data_at_delete: true

Top ∆

3.6 Fetching Auditing information from the application via REST API

3.6.1 Create a bean of AuditReader

Hibernate Envers provides two methods to fetch the audit information for an entity.

• forRevisionsOfEntity – This is used to fetch all revision changes that an entity has went through.
• forRevisionsOfEntityWithChanges – This is to fetch revision changes along with information on which fields and data was changed in each revision.

You will need an instance of class org.hibernate.envers.AuditReader, to create AuditQuery that will be used to fetch audit information

Add following configuration class to create a bean of org.hibernate.envers.AuditReader

@Configurationpublic class AuditConfiguration {    private final EntityManagerFactory entityManagerFactory;    AuditConfiguration(EntityManagerFactory entityManagerFactory) {        this.entityManagerFactory = entityManagerFactory;    }    @Bean    AuditReader auditReader() {        return AuditReaderFactory.get(entityManagerFactory.createEntityManager());    }}

3.6.2 Create a method in Customer Service to fetch Customer audits

Now that we have create a bean of AuditReader, go ahead and autowire the AuditReader dependency into the CustomerService class

public class CustomerService {
    ...
    ...
    @Autowired
    private AuditReader auditReader;
    ...
}

Now add a method in CustomerService, to fetch audit revision details for customer entity

public List<?> getRevisions(Long id, boolean fetchChanges) {
        AuditQuery auditQuery = null;

        if(fetchChanges) {
            auditQuery = auditReader.createQuery()
                    .forRevisionsOfEntityWithChanges(Customer.class, true);
        }
        else {
            auditQuery = auditReader.createQuery()
                    .forRevisionsOfEntity(Customer.class, true);
        }
        auditQuery.add(AuditEntity.id().eq(id));
        return auditQuery.getResultList();
    }

This method will fetch only revisions if fetchChanges boolean is false, else it will return all revisions with details of what has been changed.

3.6.3 Expose an Rest API for Customer Audits

Expose a Rest API like this

@GetMapping(path = "/{id}/revisions", produces = MediaType.APPLICATION_JSON_VALUE)
public ResponseEntity<?> getRevisions(@PathVariable(name = "id") String customerId,
           @RequestParam(value = "fetchChanges", required = false) boolean fetchChanges) {
        
    List results = customerService.getRevisions(Long.valueOf(customerId), fetchChanges);
    return ResponseEntity.ok(results);
}

Now lets hit the APIs to fetch the audit information. Hit the below URL to get audit revisions for customer Id 2.

curl -i -u admin02@tw.com:admin02@123# \
  -X GET -H "Content-Type: application/json" \
   http://localhost:8080/customer/2/revisions

You will get below error, as application is not able to serialize the AuditRevisionEntity

com.fasterxml.jackson.databind.exc.InvalidDefinitionException: No serializer found for class org.hibernate.proxy.pojo.bytebuddy.ByteBuddyInterceptor and no properties discovered to create BeanSerializer

To fix this, add following annotation to AuditRevisionEntity class

@JsonIgnoreProperties({"hibernateLazyInitializer", "handler"})
public class AuditRevisionEntity extends DefaultRevisionEntity {
   ....
}

Now if you hit the same URL back again, you should get list of revisions for customer with id = 2 (as shown below)

[
  {
    "id": 1,
    "timestamp": 1608084027455,
    "user": "admin01@tw.com",
    "revisionDate": "2020-12-16T02:00:27.455+0000"
  },
  {
    "id": 2,
    "timestamp": 1608084069397,
    "user": "admin02@tw.com",
    "revisionDate": "2020-12-16T02:01:09.397+0000"
  }
]

3.6.4 Fetching audits with change details

In above example we only got information about when the object was changed. We did not get information of who changed it.

To do that hit the same URL with query param fetchChanges=true

curl -i -u admin02@tw.com:admin02@123# \
  -X GET -H "Content-Type: application/json" \
   http://localhost:8080/customer/2/revisions\?fetchChanges\=true

Unfortunately you will get following error

{
  "statusCode": 400,
  "errorMessage": "The specified entity [com.dev.springdemo.customer.model.Customer] does not support or use modified flags."
}

This is because we have not configured our Customer Entity to track which fields were changed.

Modify the @Audited annotation on Customer Entity , and add property withModifiedFlag = true

@Audited(withModifiedFlag = true)
public class Customer implements Serializable {
   ...
}

Now if you re-run the code, you will again get error (something like below)

Schema-validation: missing column [email_address_mod] in table [customer_audit]

Envers can track field changes by adding _mod column for every entity field in *_audit table.

So for every field of customer, we need to add a corresponding boolean column in customer_audit table and suffix the column name with _mod

E.g. for email_address column in customer table, add a email_address_mod field in the customer_audit table.

So for adding all *_mod columns in customer_audit table, run following sql script.

ALTER TABLE `customer_audit`
ADD COLUMN `created_by_mod` TINYINT(1) NULL,
ADD COLUMN `created_on_mod` TINYINT(1) NULL ,
ADD COLUMN `email_address_mod` boolean NULL,
ADD COLUMN `first_name_mod` boolean NULL,
ADD COLUMN `last_name_mod` boolean NULL,
ADD COLUMN `updated_by_mod` boolean NULL,
ADD COLUMN `updated_on_mod` boolean NULL;

Now we are all set to test it out finally. Create some new audit entries by creating and updating the customers.
Now if you fetch revision with chagnes, you will get an output something like this.

[
  [
    {
      "id": 4,
      "firstName": "Clark",
      "lastName": "Kent",
      "emailAddress": "superman@gmail.com",
      "addresses": null,
      "createdBy": "admin01@tw.com",
      "updatedBy": "admin01@tw.com",
      "createdOn": "2020-12-16T03:25:01.000+0000",
      "updatedOn": "2020-12-16T03:25:01.000+0000"
    },
    {
      "id": 3,
      "timestamp": 1608089100811,
      "user": "admin01@tw.com",
      "revisionDate": "2020-12-16T03:25:00.811+0000"
    },
    "ADD",
    []
  ],
  [
    {
      "id": 4,
      "firstName": "Super",
      "lastName": "Man",
      "emailAddress": "superman@gmail.com",
      "addresses": null,
      "createdBy": "admin01@tw.com",
      "updatedBy": "admin02@tw.com",
      "createdOn": "2020-12-16T03:25:01.000+0000",
      "updatedOn": "2020-12-16T03:25:49.000+0000"
    },
    {
      "id": 4,
      "timestamp": 1608089148899,
      "user": "admin02@tw.com",
      "revisionDate": "2020-12-16T03:25:48.899+0000"
    },
    "MOD",
    [
      "firstName",
      "lastName",
      "updatedBy",
      "updatedOn"
    ]
  ]
]