- Posts: 4
Discrepancy between PySAM and UI results
- Sunny Day
- Topic Author
Less
More
21 Dec 2024 23:06 #13739
by Sunny Day
Discrepancy between PySAM and UI results was created by Sunny Day
Hello
I've encountered a scenario where there is a significant difference between the output PySAM and the SAM UI.
To reproduce the error:
1. Open attached SAM file "GenericBatterySingleOwner_6b.sam". This is essentially a default configuration of Generic Battery Single Owner model with some modifications to the battery sizing, battery dispatch and Time of Delivery.
2. Export SAM UI outputs: In the Simulation results area: Data tables >> Save as CSV. Save as "sam_ui_output_6b.csv"
3. Export inputs for pysam: Generate Code >> PySAM JSON Save to the same directory as used for step 2.
To compare the results, I ran the following script in python (be sure to enter the correct value for the local directory where the data was saved)
The significant discrepancy seems to first arise on the 30th of April. The SOC per pySAM does not decrease after the battery discharges to the grid in the evening. The UI data seems better, with a corresponding reduction in SOC after the discharge.
Is this a pysam bug or is there something that I'm missing?
Many thanks!
I've encountered a scenario where there is a significant difference between the output PySAM and the SAM UI.
To reproduce the error:
1. Open attached SAM file "GenericBatterySingleOwner_6b.sam". This is essentially a default configuration of Generic Battery Single Owner model with some modifications to the battery sizing, battery dispatch and Time of Delivery.
2. Export SAM UI outputs: In the Simulation results area: Data tables >> Save as CSV. Save as "sam_ui_output_6b.csv"
3. Export inputs for pysam: Generate Code >> PySAM JSON Save to the same directory as used for step 2.
To compare the results, I ran the following script in python (be sure to enter the correct value for the local directory where the data was saved)
Code:
[color=#0033b3]import [/color]json
[color=#0033b3]import [/color]pandas [color=#0033b3]as [/color]pd
[color=#0033b3]import [/color]PySAM.Battery [color=#0033b3]as [/color]Battery
[color=#0033b3]import [/color]PySAM.GenericSystem [color=#0033b3]as [/color]GenericSystem
[color=#0033b3]import [/color]PySAM.Grid [color=#0033b3]as [/color]Grid
[color=#0033b3]import [/color]PySAM.Singleowner [color=#0033b3]as [/color]SingleOwner
[color=#0033b3]import [/color]PySAM.Utilityrate5 [color=#0033b3]as [/color]UtilityRate
[color=#0033b3]import [/color]os
[color=#0033b3]from [/color]numpy.testing [color=#0033b3]import [/color]assert_allclose
[color=#0033b3]import [/color]numpy [color=#0033b3]as [/color]np
[color=#0033b3]from [/color]pathlib [color=#0033b3]import [/color]Path
[color=#0033b3]import [/color]matplotlib.pyplot [color=#0033b3]as [/color]plt
[color=#8c8c8c]# directory where default inputs are stored
[/color]working_dir = Path(r'<specify directory of saved files here!!!!!!!>')
# open SAM UI result
sam_ui_result = pd.read_csv(os.path.join(working_dir, [color=#067d17]"sam_ui_output_6b.csv"[/color]))
[color=#8c8c8c]# Create a list of filenames. The order must be the same as the execution order
[/color]module_names = [
[color=#067d17]"generic_system"[/color],
[color=#067d17]"battery"[/color],
[color=#067d17]"grid"[/color],
[color=#067d17]"utilityrate5"[/color],
[color=#067d17]"singleowner"[/color],
]
file_names = [
[color=#067d17]"GenericBatterySingleOwner_6b_" [/color]+ x + [color=#067d17]".json" [/color][color=#0033b3]for [/color]x [color=#0033b3]in [/color]module_names
]
paths = [os.path.join(working_dir, file_name) [color=#0033b3]for [/color]file_name [color=#0033b3]in [/color]file_names]
[color=#8c8c8c]# read files
[/color]module_inputs = []
[color=#0033b3]for [/color]f [color=#0033b3]in [/color]paths:
[color=#0033b3]with [/color][color=#000080]open[/color](f, [color=#067d17]"r"[/color]) [color=#0033b3]as [/color]file:
[color=#0033b3]try[/color]:
module_inputs.append(json.load(file))
[color=#0033b3]except [/color][color=#000080]IOError [/color][color=#0033b3]as [/color]e:
[color=#0033b3]raise [/color][color=#000080]IOError[/color]([color=#067d17]"Error when reading SAM json files" [/color]+ [color=#000080]str[/color](e))
[color=#8c8c8c]# initialize the primary module (Generic System)
[/color]gs = GenericSystem.new()
[color=#8c8c8c]# initialize the subsequent modules, linking them to the primary module
[/color]batt = Battery.from_existing(gs)
gr = Grid.from_existing(gs)
ur = UtilityRate.from_existing(gs)
so = SingleOwner.from_existing(gs)
[color=#8c8c8c]# Create a list of modules. The execution order must be correct
[/color]modules = [gs, batt, gr, ur, so]
[color=#8c8c8c]# transfer input from module_inputs list to the modules
[/color][color=#0033b3]for [/color]m, data [color=#0033b3]in [/color][color=#000080]zip[/color](modules, module_inputs):
[color=#8c8c8c]# Loop through each key-value pair
[/color][color=#8c8c8c] [/color][color=#0033b3]for [/color]k, v [color=#0033b3]in [/color]data.items():
[color=#0033b3]if [/color]k != [color=#067d17]"number_inputs"[/color]:
k = k.replace(
[color=#067d17]"adjust_"[/color], [color=#067d17]""
[/color][color=#067d17] [/color]) [color=#8c8c8c]# workaround for known bug, see https://github.com/NREL/pysam/issues/164
[/color][color=#8c8c8c] [/color][color=#0033b3]try[/color]:
m.value(k, v)
[color=#0033b3]except [/color][color=#000080]Exception [/color][color=#0033b3]as [/color]e: [color=#8c8c8c]# Catch any exception
[/color][color=#8c8c8c] [/color][color=#0033b3]raise [/color][color=#000080]Exception[/color](
[color=#067d17]f"Error with key: [/color][color=#0037a6]{[/color]k[color=#0037a6]}[/color][color=#067d17], value: [/color][color=#0037a6]{[/color]v[color=#0037a6]}[/color][color=#067d17]. Original error: [/color][color=#0037a6]{[/color]e[color=#0037a6]}[/color][color=#067d17]"
[/color][color=#067d17] [/color])
[color=#8c8c8c]# execute pySAM modules in sequence
[/color][color=#0033b3]for [/color]m [color=#0033b3]in [/color]modules:
m.execute()
[color=#8c8c8c]## plotting
[/color]
sam_ui_result[[color=#067d17]'pysam_gen'[/color]] = modules[[color=#1750eb]1[/color]].SystemOutput.gen
sam_ui_result[[color=#067d17]'pysam_ppa_price'[/color]] = modules[[color=#1750eb]1[/color]].Outputs.market_sell_rate_series_yr1
sam_ui_result[[color=#067d17]'pysam_soc'[/color]] = modules[[color=#1750eb]1[/color]].Outputs.batt_SOC
datetimes_1_year = pd.date_range([color=#660099]start[/color]=[color=#067d17]'1990-1-1 0:30:00'[/color], [color=#660099]periods [/color]= [color=#1750eb]8760[/color], [color=#660099]freq [/color]= [color=#067d17]'H'[/color])
sam_ui_result[[color=#067d17]'timestamp'[/color]] = [color=#000080]list[/color](datetimes_1_year)*[color=#1750eb]25
[/color]year = []
[color=#0033b3]for [/color]i [color=#0033b3]in [/color]np.arange([color=#1750eb]25[/color]):
year = year + [i]*[color=#1750eb]8760
[/color]sam_ui_result[[color=#067d17]'year'[/color]] = year
sam_ui_result[[color=#067d17]'month'[/color]] = sam_ui_result[[color=#067d17]'timestamp'[/color]].dt.month
sam_ui_result[[color=#067d17]'day'[/color]] = sam_ui_result[[color=#067d17]'timestamp'[/color]].dt.day
sam_ui_result[[color=#067d17]'disc'[/color]] = [color=#000080]abs[/color](
sam_ui_result[[color=#067d17]"System power generated | (kW)"[/color]]
- sam_ui_result[[color=#067d17]'pysam_gen'[/color]]
)
max_disc = sam_ui_result.loc[sam_ui_result[[color=#067d17]'year'[/color]]==[color=#1750eb]0[/color],[color=#067d17]'disc'[/color]].idxmax()
plot_year_wc, plot_month_wc, plot_day_wc = sam_ui_result.loc[max_disc, [[color=#067d17]'year'[/color],[color=#067d17]'month'[/color],[color=#067d17]'day'[/color]]]
# the date of the worst-case discrepancy was May 1st
# plot for days and months surrounding the worst-case date
[color=#0033b3]for [/color]plot_month [color=#0033b3]in [/color][[color=#1750eb]4[/color], [color=#1750eb]5[/color]]:
[color=#0033b3]for [/color]plot_day [color=#0033b3]in [/color][color=#000080]range[/color]([color=#1750eb]1[/color],[color=#1750eb]31[/color]):
plot_df = sam_ui_result.loc[(
(sam_ui_result[[color=#067d17]'year'[/color]] == plot_year)
& (sam_ui_result[[color=#067d17]'month'[/color]] == plot_month)
& (sam_ui_result[[color=#067d17]'day'[/color]] == plot_day)
),
:].copy()
fig, axs = plt.subplots([color=#1750eb]3[/color],[color=#1750eb]1[/color], [color=#660099]sharex[/color]=[color=#0033b3]True[/color], [color=#660099]figsize[/color]=([color=#1750eb]10[/color],[color=#1750eb]10[/color]))
[color=#0033b3]for [/color]col [color=#0033b3]in [/color][[color=#067d17]'System power generated | (kW)'[/color],
[color=#067d17]'Electricity to grid from battery AC | (kW)'[/color],
[color=#067d17]'Electricity to battery from grid AC | (kW)'[/color],
[color=#067d17]'pysam_gen'[/color]]:
axs[[color=#1750eb]0[/color]].plot(plot_df[[color=#067d17]'timestamp'[/color]], plot_df[col], [color=#660099]label [/color]= col, [color=#660099]marker[/color]=[color=#067d17]'o'[/color])
axs[[color=#1750eb]1[/color]].plot(plot_df[[color=#067d17]'timestamp'[/color]], plot_df[[color=#067d17]'Power price for battery dispatch | ($/MWh)'[/color]], [color=#660099]marker[/color]=[color=#067d17]'o'[/color], [color=#660099]label[/color]=[color=#067d17]'UI'[/color])
axs[[color=#1750eb]1[/color]].plot(plot_df[[color=#067d17]'timestamp'[/color]], plot_df[[color=#067d17]'pysam_ppa_price'[/color]], [color=#660099]marker[/color]=[color=#067d17]'o'[/color], [color=#660099]label[/color]=[color=#067d17]'pysam'[/color])
axs[[color=#1750eb]1[/color]].set_ylabel([color=#067d17]'Power price for battery dispatch | ($/MWh)'[/color])
axs[[color=#1750eb]2[/color]].plot(plot_df[[color=#067d17]'timestamp'[/color]], plot_df[[color=#067d17]'Battery state of charge | (%)'[/color]], [color=#660099]marker[/color]=[color=#067d17]'o'[/color], [color=#660099]label[/color]=[color=#067d17]'UI'[/color])
axs[[color=#1750eb]2[/color]].plot(plot_df[[color=#067d17]'timestamp'[/color]], plot_df[[color=#067d17]'pysam_soc'[/color]], [color=#660099]marker[/color]=[color=#067d17]'o'[/color], [color=#660099]label[/color]=[color=#067d17]'pysam'[/color])
axs[[color=#1750eb]2[/color]].set_ylabel([color=#067d17]'Battery state of charge | (%)'[/color])
axs[[color=#1750eb]0[/color]].legend([color=#660099]prop[/color]={[color=#067d17]'size'[/color]: [color=#1750eb]8[/color]})
axs[[color=#1750eb]1[/color]].legend()
axs[[color=#1750eb]2[/color]].legend()
fig.autofmt_xdate()
plt.title([color=#067d17]f'year[/color][color=#0037a6]{[/color]plot_year[color=#0037a6]}[/color][color=#067d17]_month[/color][color=#0037a6]{[/color]plot_month[color=#0037a6]}[/color][color=#067d17]_day[/color][color=#0037a6]{[/color]plot_day[color=#0037a6]}[/color][color=#067d17]'[/color])
plt.savefig(os.path.join(working_dir,
[color=#067d17]f'troubleshoot_case6b_year[/color][color=#0037a6]{[/color]plot_year[color=#0037a6]}[/color][color=#067d17]_month[/color][color=#0037a6]{[/color]plot_month[color=#0037a6]}[/color][color=#067d17]_day[/color][color=#0037a6]{[/color]plot_day[color=#0037a6]}[/color][color=#067d17].jpg'[/color]), [color=#660099]dpi[/color]=[color=#1750eb]300[/color], [color=#660099]bbox_inches[/color]=[color=#067d17]'tight'[/color])
[color=#8c8c8c]# plt.show()
[/color][color=#8c8c8c] [/color]plt.close()
# the following raises an assertion error
assert_allclose(
modules[[color=#1750eb]1[/color]].SystemOutput.gen[:],
sam_ui_result[[color=#067d17]"System power generated | (kW)"[/color]][:],
[color=#660099]rtol[/color]=[color=#1750eb]1e-3[/color],
[color=#660099]atol[/color]=[color=#1750eb]0.1[/color],
)
The significant discrepancy seems to first arise on the 30th of April. The SOC per pySAM does not decrease after the battery discharges to the grid in the evening. The UI data seems better, with a corresponding reduction in SOC after the discharge.
Is this a pysam bug or is there something that I'm missing?
Many thanks!
Attachments:
Please Log in or Create an account to join the conversation.
- Paul Gilman
Less
More
- Posts: 5474
06 Jan 2025 12:59 #13784
by Paul Gilman
Replied by Paul Gilman on topic Discrepancy between PySAM and UI results
Hello,
A common reason for discrepancies between SAM and PySAM is differences in inputs or versions (the SSC version of SAM and PySAM should match).
If you would like me to help troubleshoot, please attach a minimum reproducible example of Python that demonstrates the discrepancy, ideally without the code for plotting and without the color tags.
Best regards,
Paul.
A common reason for discrepancies between SAM and PySAM is differences in inputs or versions (the SSC version of SAM and PySAM should match).
If you would like me to help troubleshoot, please attach a minimum reproducible example of Python that demonstrates the discrepancy, ideally without the code for plotting and without the color tags.
Best regards,
Paul.
Please Log in or Create an account to join the conversation.
- Sunny Day
- Topic Author
Less
More
- Posts: 4
06 Jan 2025 15:57 #13786
by Sunny Day
Replied by Sunny Day on topic Discrepancy between PySAM and UI results
Thanks Paul. Yes indeed, upon inspection the SSC versions do not match. I'm running NREL-PySAM v 5.0.0 (SSC version 288) and the SAM UI is 2023.12.17r2 (SSC version 292). I am using the SAM UI to validate my implementation of PySAM, and it would be preferable to find a version of the SAM UI with a SSC version that matches the version of the PySAM version I'm using (and not the other way around). However of the available UI versions listed on [url]
sam.nrel.gov/download.html
[/url] I do not see one that has the right SSC version (the previous version of the UI is 2022.11.21r3 with SSC version 280). Does this mean it is not possible to make a direct comparison here?
Attached is a minimum reproducible example that shows the discrepancy when running NREL-PySAM v 5.0.0 (SSC version 288) using files generated with SAM UI 2023.12.17r2 (SSC version 292). I apologize about the color tags in my last post, I suspected they were added automatically when I uploaded the code into the forum portal.
Attached is a minimum reproducible example that shows the discrepancy when running NREL-PySAM v 5.0.0 (SSC version 288) using files generated with SAM UI 2023.12.17r2 (SSC version 292). I apologize about the color tags in my last post, I suspected they were added automatically when I uploaded the code into the forum portal.
Attachments:
Please Log in or Create an account to join the conversation.
- Paul Gilman
Less
More
- Posts: 5474
07 Jan 2025 17:45 #13788
by Paul Gilman
Replied by Paul Gilman on topic Discrepancy between PySAM and UI results
Hello,
Sorry about the version discrepancies between SAM and PySAM.
PySAM 5.0.0 and SAM 2023.12.17 (original release) both use SSC 288. Only SAM 2023.12.17 r2 is available for download from the SAM website because it is the most recent update to the original release of SAM 2023.12.17.
You can use this link to download the original release version of SAM 2023.12.17 for the purpose of your comparison.
Please note that PySAM 5.0.0 and SAM 2023.12.17 are both out-of-date versions, so I would recommend eventually using the latest versions.
You can install and run multiple versions of SAM on your computer, but be aware that once you save a .sam file in a given version of SAM, you will no longer be able to use an older version to open it. It may be a good idea to save different copies of your .sam files for different versions of the software.
Let me know if you continue to see differences in results between PySAM 5.0.0 and SAM 2023.12.17 original release.
Best regards,
Paul.
Sorry about the version discrepancies between SAM and PySAM.
PySAM 5.0.0 and SAM 2023.12.17 (original release) both use SSC 288. Only SAM 2023.12.17 r2 is available for download from the SAM website because it is the most recent update to the original release of SAM 2023.12.17.
You can use this link to download the original release version of SAM 2023.12.17 for the purpose of your comparison.
Please note that PySAM 5.0.0 and SAM 2023.12.17 are both out-of-date versions, so I would recommend eventually using the latest versions.
You can install and run multiple versions of SAM on your computer, but be aware that once you save a .sam file in a given version of SAM, you will no longer be able to use an older version to open it. It may be a good idea to save different copies of your .sam files for different versions of the software.
Let me know if you continue to see differences in results between PySAM 5.0.0 and SAM 2023.12.17 original release.
Best regards,
Paul.
Please Log in or Create an account to join the conversation.
- Sunny Day
- Topic Author
Less
More
- Posts: 4
09 Jan 2025 13:30 #13801
by Sunny Day
Replied by Sunny Day on topic Discrepancy between PySAM and UI results
Hi Paul
I installed 2023.12.17, original release. Repeating the comparison - now that the SSC versions are the same - I see agreement. Thank you very much for your help!
I installed 2023.12.17, original release. Repeating the comparison - now that the SSC versions are the same - I see agreement. Thank you very much for your help!
Please Log in or Create an account to join the conversation.
Moderators: Paul Gilman