People enjoyed Herbert's company—his powers of conversation were considered remarkable—but he had a dark side. When he drank, he was often quarrelsome, and when sober, he could become enraged by a fancied insult. He was involved in at least three duels. One challenge came after he called a man a liar in a barroom argument. A snowstorm prevented Herbert from reaching the dueling ground on time, and he returned home to learn that his adversary's second was accusing him of cowardice. Herbert found the second in a bar and fired twice at him as the man fled out the door. The balls missed, but the Herald, a rival of Herbert's paper, asked its readers to visit the bar and see the "two holes made in a republican door by the royal blood of Plantagenet."
In 1839 Herbert met and married 18-year-old Sarah Barker, whose father was the mayor of Bangor, Maine. In 1841 she gave birth to a son, William George, and the next year she bore a daughter, Louisa. But the often sickly Sarah died in 1843, and shortly afterward so did Louisa, a victim of cholera.
Herbert's father then made him an offer of partial reconciliation. In exchange for sending his grandson to live in England, the elder Herbert gave his exiled son the money to buy a home. The younger Herbert bought an acre in New Jersey on the banks of the Passaic River near Newark. There, bordering on Mount Pleasant Cemetery, he built a Tudor cottage he named The Cedars.
As time passed, Herbert's behavior became more and more eccentric. Still wanting to be Herbert but hungering to be recognized in daily life as Forester, he often went about in a shooting jacket, boots and a fur cap, a hunting dog trailing at his heels. After several of his friends and his father died, Herbert became almost a recluse. Then in February 1858 he married Adela R. Budlong of Providence, whom he claimed to have met when he rescued her during a riot in New York City only three weeks earlier.
The marriage lasted a month or two. Using the pretext that she wanted to visit her mother, Adela took off. Several weeks later Herbert got confirmation that his wife had left him for good when he received a legal notice from an Indiana court. It asked him to show cause why Adela R. Budlong, his wife, should not be granted a divorce on the ground of cruelty.
The night after receiving the letter, Herbert invited a number of friends to dine with him in his suite at the Stevens House, a Manhattan hotel where he had been living since Adela's departure. Only one, Philip Anthon, showed up. After dinner Herbert left Anthon and went into his room, where he shot himself in the heart. Staggering back into the reception chamber, he exclaimed to Anthon, "I told you I would do it!" and fell to the floor.
Herbert was buried in the cemetery next to The Cedars, but his relatives in England refused to buy a tombstone. Eighteen years later a group of Newark citizens raised the money for a stone, and on it they had inscribed the words that Herbert had requested in a suicide note: "Henry William Herbert of England, Aged 51 Years." Directly below is the one-word epitaph that Herbert also chose for himself, Infelicissimus, which is Latin for "most unfortunate."