Custom Group: You will need to manually add references to the group.
A Smart Group: You will need to define your search terms so that the references will be added or updated automatically within your EndNote Library.
A Group Set: A group set serves as the name of a header. You can create Custom Groups and Smart Groups under a Group Set.
Note: Smart Groups and Group Sets do not show up in EndNote Online, however they do sync to your other desktop EndNote installations.
This first video demonstrates how to create a Custom Group, a Smart Group, and a Group Set.
This 2nd video discusses smart and combination groups.