Mfrhtyj

Given two (numeric) samples I would like to show that there is not a significant difference between the two means $mu_{1}$ and $mu_{2}$.

If my goal was to show a significant difference I would formulate the $t$-test as follows:

(1) $H_{0}: mu_{1} = mu_{2}$ vs $H_{1}: mu_{1} neq mu_{2}$

I learned in school that the null hypothesis should always represent the "common" belief and the alternative hypothesis should represent the change that I would like to show.

So then if my goal is to show that there is not a significant difference between both means, should I formulate the test like this?

(2) $H_{0}: mu_{1} neq mu_{2}$ vs $H_{1}: mu_{1} = mu_{2}$

Or can I use the first test (1) and when I am not able to reject the null hypothesis say that there is not a significant difference?

You cannot use the first test in the way you describe, because failure to reject in the first test only says that you were unable to reject $H_0$ nothing more than that. It is like only being given the information that "the prosecutor was unable to provide the jury with enough evidence to secure a conviction" - that does not tell you that the suspect is innocent.

The second test is not usable in practice, because no matter how much data you have, you cannot exclude the possibility of very small differences.

What you can do is to look at
$$H_{0}: |mu_{1} - mu_{2}|>delta text{ vs }H_{1}: |mu_{1} - mu_{2}| leq delta,$$
i.e. try to reject the null hypothesis that the absolute size of the difference is greater than some difference $delta>0$. $delta$ would be chosen e.g. so that any difference smaller than that is for all (or your specific) practical purposes irrelevant.

I learned in school that the null hypothesis should always represent the "common" belief and the alternative hypothesis should represent the change that I would like to show.

That is not accurate explanation of the null hypothesis. The null hypothesis is simply a hypothesis that consists of a specific distribution from which probabilities can be calculated. The reason we use $mu_1=mu_2$ as the null hypothesis has nothing to do with whether this is the "common" belief. It's used as the null hypothesis because if we hypothesize that the mean is a specific value, then given a particular set of data we can calculate the probability of seeing that data. We can't use $mu neq mu_2$ as our null hypothesis because there's no way to calculate p-values based simply on the hypothesis that the means aren't equal to a particular value. Consider the following problem:

The weights of apples have a standard deviation of 5 grams. The mean is not equal to 100. What is the probability of seeing an apple with a weight of 110 grams?

There's no way to answer that, because simply being told what the mean isn't is not enough to calculate probabilities.

Björn suggests testing the hypothesis that the difference in means is greater than some $delta_0$. How that would work is to take the null hypothesis as that the difference is equal to exactly $delta_0$. Then once you have the data, you can calculate the p-value given that $delta_0$. Call that $p_{delta_0}$. If the difference in sample means is less than $delta_0$, then the the p-value would have been even smaller than $p_{delta_0}$ if we had chosen $delta$ to be larger than $delta_0$. We reject the null if the p-value is less than $alpha$, so if we're rejecting under that null, that means that $p_{delta_0} < alpha$. And since $p_{delta}<p_{delta_0}$ for any $delta>delta_0$, we can conclude that $p_{delta}<alpha$ for any $delta>delta_0$. Thus, we can not only reject this null of $delta_0$, but we can reject any null with a larger $delta$. It is only because of this ability to get an upper bound on p that we don't need a specific value for $delta$. If we just take "$delta$ is larger than zero" as our null hypothesis, without any lower bound for $delta$, then there is no upper bound for p, and so we cannot conclude that it is lower than $alpha$.