@UniqueConstraint and @Column(unique = true) in hibernate annotation

As said before, @Column(unique = true) is a shortcut to UniqueConstraint when it is only a single field.

From the example you gave, there is a huge difference between both.

@Column(unique = true)
@ManyToOne(optional = false, fetch = FetchType.EAGER)
private ProductSerialMask mask;

@Column(unique = true)
@ManyToOne(optional = false, fetch = FetchType.EAGER)
private Group group;

This code implies that both mask and group have to be unique, but separately. That means that if, for example, you have a record with a mask.id = 1 and tries to insert another record with mask.id = 1, you'll get an error, because that column should have unique values. The same aplies for group.

On the other hand,

@Table(
   name = "product_serial_group_mask", 
   uniqueConstraints = {@UniqueConstraint(columnNames = {"mask", "group"})}
)

Implies that the values of mask + group combined should be unique. That means you can have, for example, a record with mask.id = 1 and group.id = 1, and if you try to insert another record with mask.id = 1 and group.id = 2, it'll be inserted successfully, whereas in the first case it wouldn't.

If you'd like to have both mask and group to be unique separately and to that at class level, you'd have to write the code as following:

@Table(
        name = "product_serial_group_mask",
        uniqueConstraints = {
                @UniqueConstraint(columnNames = "mask"),
                @UniqueConstraint(columnNames = "group")
        }
)

This has the same effect as the first code block.