From what I understand from my cousin Walter who is the grandson of Matthew, they had the 5 children which Matthew acknowledged all of them but with the laws of the time, he couldn't give them his surname nor could he live with them unless they were servants, that's the story I was told anyway.
This is a picture of Mary Rush.
Then when Harriet died, Matthew took up with Mary Rush, in the 1910 census Mary is listed as mulatto, living next door to William Davenport, Matthew is 27 years old, Mary Rush is 16 years old she's a servant. I didn't find them in the 1920 census so far. By the 1930 census Mary is still living next door to Matthew but she now has 4 children, her mother Emma is still living with her, William B. Davenport has since died and Matthew is still single living in the home. Walter said that Mary was a servant for show but that Matthew loved all his children and treated them as such.
Walter tells a story about going down to Porterville, Mississippi to visit the family and to see Mary Rush when he arrives, no one is home so he sits on the steps to wait, while waiting he sees some sort of procession in the street, this was in 1977. He gets up and goes towards the procession, and asks one of the people what was going on, they replied its a funeral, Walter asks for who, they told him it was for Mary Rush. An amazing story, he didn't know Mary had died but went to visit only to find out that she had passed away and he arrived just in time for the funeral.
I love to hear stories.