If org-agenda-start-on-weekday is set to an integer (by default it's set to 1, corresponding to Monday), and org-agenda-span is set to either 'week or 7, org will always start the agenda on the day specified by org-agenda-start-on-weekday.
So to always start yesterday, you must set org-agenda-start-day to "-1d" and do one of the following:
Set org-agenda-start-on-weekday to nil. Then, the first day of the agenda will be determined by org-agenda-start-day.
Set org-agenda-span to a value that isn't 'week. For example, setting it to 5 will show five days, and then it will respect org-agenda-start-day.
I recommend doing both, in case you decide you want to show a full week. Here's code to show five days starting yesterday:
(setq org-agenda-start-day "-1d")
(setq org-agenda-span 5)
(setq org-agenda-start-on-weekday nil)