Simple Summary:& nbsp;The composition of milk fat has serious importance for consumer health, and unsaturated and especially polyunsaturated fatty acids are preferred. Thus, methods of influencing the fatty acid profile of milk fat are intensively being studied, both genetically and nongenetically. This paper analyzed the effects of polymorphisms in some genes, breeds, lactation parity and stage, and farm on the milk fatty acid profile. The farm was the most significant factor influencing the profile, followed by the lactation stage and their interaction. The lactation parity did not show much importance. The effect of the cow’s breed was minimal. Considering that all 49 individual fatty acids and 11 groups were evaluated, the influence of the gene polymorphisms mentioned was not strong. SCD1 showed significance in eleven cases.& nbsp;This study aimed to analyze the factors affecting the fatty acid (FA) profile in cow’s milk. The effects of a farm, lactation parity and stage, breed and polymorphisms in the AGPAT6, DGAT1, LEP, FASN and SCD1 genes were evaluated. A total of 196 Holstein cows, 226 Simmental cows and seven crosses were sampled 751 times. The cows were kept at five farms and were in the first up to the sixth lactation, and 49 individual FAs and 11 groups were analyzed. The farm significantly affected the proportion of all FAs except for C16:1n-7c and isoC14:0. Additionally, the lactation stage was significant for most FAs, and the opposite was true for lactation parity. The effect of the breed was negligible. For the gene polymorphisms, the SCD1 TT genotype exceeded the CC in C10:0, C12:0, C14:0, C16:1n-7c and C18:2, and the opposite was true for C10:1, C12:1, C14:1n-5c, isoC17:0, C16:1 and C18:1, i.e., the TT genotype was higher for saturated FAs, and the CT genotype was higher for monounsaturated FAs. The results hint at the intermediary heredity of the SCD1 gene. The FASN gene was strongly associated with four FAs and branched-chain FAs, and genotype AG was better than GG. LEP was significant for five individual FAs and branched-chain FAs. The differences in FA composition among genotypes were rather small, which could lead to overestimation of the effect and needs to be considered in the next research.