Actually, it's much more complicated than that.
Nothing stopping you from elaborating.
For one, there's obviously genetics involved.
Also, how often you eat, when you eat, and what you eat.
Your endocrine glands affect your hormone levels. Their functioning also make a difference in your weight. Fun fact, contrary to popular belief, adipose tissue (body fat) isn't merely for energy storage, it is actually the largest endocrine gland in the human body.
Then there's makeup of your gut microbiome. There are billions of microorganisms in your digestive tract of several different types. In fact if they were not there you couldn't digest food. The proportion of the different types of microorganisms can also affect your weight.
There's also medication. Many medications including but not limited to antacids, anabolic steroids, statins, many pain medications and medications for depression and anxiety disorders can affect your weight.
Then there's exercise, not only the amount of exercise you get, but also the type of exercise. Some types of exercise will help you gain more mass than others.