I'm afraid that spenlinhauer is right in that the exact same line (when using the "APPENDDATE" argument with custom format) doesn't work via a .BAT or .CMD file, but it does work when you execute the line directly form a CMD window.
However, there's a solution for you, spenlinhauer: create a shortcut for your "BRC32.exe" file. Then open it's properties and add the following to the "Target" field (with a space between "...BRC32.exe" and "/APPENDDATE..."):/APPENDDATE:M:P:" ":-:10:"%Y-%m-%d %I'%M%p" /PATTERN:"backup*.log" /FIXEDEXT:".txt" /EXECUTE
Then use that shortcut to rename. That should do the trick